Language, Embodiment and Social Intelligence

It is an honor to have this chance to tie together themes from my recent research, and to sketch some challenges and opportunities for NLG in face-to-face conversational interaction. Communication reflects our general involvement in one anothers’ lives. Through the choices we manifest with one another, we share our thoughts and feelings, strengthen our relationships and further our joint projects. We rely not only on words to articulate our perspectives, but also on a heterogeneous array of accompanying efforts: embodied deixis, expressive movement, presentation of iconic imagery and instrumental action in the world. Words showcase the distinctive linguistic knowledge which human communication exploits. But people’s diverse choices in conversation in fact come together to reveal multifaceted, interrelated meanings, in which all our actions, verbal and nonverbal, fit the situation and further social purposes. In the best case, they let interlocutors understand not just each other’s words, but each other. As NLG researchers, I argue, we have good reason to work towards models of social cognition that embrace the breadth of conversation. Scientifically, it connects us to an emerging consensus in favor of a general human pragmatic competence, rooted in capacities for engagement, coordination, shared intentionality and extended relationships. Technically, it lets us position ourselves as part of an emerging revolution in integrative Artificial Intelligence, characterized by research challenges like human–robot interaction and the design of virtual humans, and applications in assistive and educational technology and interactive entertainment. Researchers are already hard at work to place our accounts of embodied action in conversation in contact with pragmatic theories derived from text discourse and spoken dialogue. In my own experience, such work proves both illuminating and exciting. For example, it challenges us to support and refine theories of discourse coherence by accounting for the discourse relations and default inference that determine the joint interpretation of coverbal gesture and its accompanying speech (Lascarides and Stone, 2008). And it challenges us to show how speakers work across modalities to engage with, disambiguate, and (on acceptance) recapitulate each others’ communicative actions, to ground their meanings (Lascarides and Stone, In Preparation). The closer we look at conversation, the more we can fit all its behaviors into a unitary framework—inviting us to implement behavioral control for embodied social agents through a pervasive analogy to NLG. We can already pursue such implementations easily. Computationally, motion is just sequence data, and we can manipulate it in parallel ways to the speech data we already use in spoken language generation (Stone et al., 2004). At a higher level, we can represent an embodied performance through a matrix of discrete actions selected and synchronized to an abstract time-line, as in our RUTH system (DeCarlo et al., 2004; Stone and Oh, 2008). This lets us use any NLG method that manipulates structured selections of discrete actions as an architecture for the production of embodied behavior. Templates, as in (Stone and DeCarlo, 2003; Stone et al., 2004), offer